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Foreword 


This evaluation of the Higher Achievement pro- 
gram was initiated by Public/Private Ventures in 
partnership with Dr. Leigh Linden, who was at 
Columbia University at the start of the project and 
now serves as an assistant professor of econom- 
ics and public affairs at the University of Texas 
at Austin. The study began in 2006, adding to an 
extensive body of research that P/PV had con- 
ducted on out-of-school-time programs over several 
decades. When P/PV ceased operations in July of 
2012, this brief was still in development, and our 
colleagues at MDRC generously agreed to publish 
it. The findings and conclusions are solely those of 
the authors. 


Staying On Track: Testing Higher Achievement’s Long-Term Impact on Academic Outcomes and High School Choice 


4 


Acknowledgments 


This research was made possible by grants from The 
Atlantic Philanthropies, Bank of America, the Smith 
Richardson Foundation, the Spencer Foundation, 
The Wallace Foundation and the William T. Grant 
Foundation. We are particularly indebted to Robert 
Granger at the William T. Grant Foundation for his 
support and guidance during the earliest phases of 
this project. 

In addition, we are grateful to the youth, parents, 
mentors, teachers and program staff who took the 
time to complete surveys or participate in our inter- 
views and focus groups. Their insights have been 
invaluable in expanding our knowledge of effective 
out-of-school-time programs. 

Staff at Higher Achievement were supportive 
partners throughout the study’s implementation. 
Numerous staff members helped set up our site 
visits, cooperated with data collection efforts, and 
patiently responded to our many requests over the 
course of the evaluation. The efforts of Edsson 
Contreras and Gail Williams (around youth recruit- 
ment) were particularly instrumental to the study’s 
success. Special recognition goes to the leadership 
of Higher Achievement. Maureen Holla helped 
design and secure initial funding for the project; 
Lynsey Jeffries and Richard Tagle provided contin- 
ued support throughout data collection. 

We also thank our partners at Survey Research 
Management (SRM). Their excellent staff, research 
know-how and tireless work collecting parent and 
youth surveys yielded strong response rates at every 
wave of data collection. Linda Kuhn, Tony Lavender 
and Rob Schroder were central in orchestrating 
SRM’s data collection efforts. 


The contributions of several other colleagues were 
also crucial to completing the project. Annelies Raue 
at Columbia University and Audrey Straus at the 
University of Texas at Austin provided exceptional 
assistance in conducting the analyses. Ama Baafra 
Abeberese, Evan Borkum and Mariesa Herrmann 
also assisted with data analysis during earlier phases 
of the project at Columbia. At P/PV, Jennifer 
McMaken co-directed the project throughout the 
first several years of its implementation. Laura 
Colket, Siobhan Cooney, Jennifer Pevar, Brittany 
Rhoades and Salem Valentino joined the project as 
summer interns, conducting site visits, analyzing the 
resulting data and assisting with survey development 
and analysis. Nora Gutierrez and Becca Raley con- 
ducted site visits and interviewed staff and mentors 
to provide implementation feedback to the program. 

Yolanda Lowler and Claudia Ross translated our sur- 
veys and consent forms. 

Chelsea Parley edited the brief, provided excellent 
feedback that shaped its direction and tone and 
coordinated its publication. Clare O’Shea provided 
final copyediting. Malish & Pagonis designed the brief. 


Staying On Track: Testing Higher Achievement’s Long-Term Impact on Academic Outcomes and High School Choice 


5 



recognized as a critical, “make or break” period 
in youth’s academic lives (Balfanz et al. 2007). 
Studies show that how well students do in middle 
school — and how smoothly they make the transi- 
tion to high school — have profound implications 
for their futures (Balfanz 2009; Grossman, Cooney 
2009; Kieffer, Marinell 2012). Attendance, grades, 
test scores and behavior during the middle grades 
all predict students’ performance in high school 
and their odds of graduating. Unfortunately, many 
students struggle academically and lose their way 
during the middle school years (Crockett et al. 

1989; Petersen, Crockett 1985). And, for youth 
who fall off track — especially in poor communities, 
where resources and supports are limited — it can be 
exceedingly difficult to get back on a steady path. 

One crucial decision that middle schoolers (and 
their families) make is where they will attend high 
school. Many districts employ school choice systems 
designed to allow students to pick a high school 
that will meet their needs and interests. Yet most 
students prefer high schools that are close to home, 
and for youth in low-income neighborhoods, this 
often means attending a more disadvantaged, lower- 
performing school (Nathanson et al. 2013). 

Youth who defy these odds and choose a competi- 
tive high school instead have much to gain. Cullen 
et al. (2005) , for instance, found that Chicago 
public middle school students who chose to attend 
a higher-achieving high school were substantially 
more likely to graduate. However, even as eighth 
graders, these students already differed in many 
ways from their peers who chose a neighborhood 
school — they had better self-reported grades and 
higher expectations for the future, felt more pre- 
pared for high school and were more likely to have 
spoken with their parents about what school to 
attend. These findings raise the question of how we 
can prepare more disadvantaged students to take 
the many steps necessary — throughout the middle 
school years — to successfully transition to a com- 
petitive, high-quality high school that can ultimately 
launch them toward college and careers. 


The Washington, DC-based Higher Achievement 
program is taking on this challenge. Higher 
Achievement targets rising fifth and sixth grad- 
ers from “at-risk communities” and serves them 
throughout the middle school years. Its goal is to 
strengthen participants’ academic skills, attitudes 
and behaviors, reinforce high aspirations and help 
students and their families navigate the process of 
applying to and selecting a high-quality high school. 

In 2006, the authors began a comprehensive multiyear 
evaluation of Higher Achievement to test its impact 
on participants’ academic performance, attitudes and 
behaviors and on their high school enrollment. The 
evaluation used random assignment — the most rigor- 
ous design available to researchers — to assess program 
impacts. This brief summarizes the study’s findings. 

About the Evaluation 

In the spring of 2006, 2007 and 2008, youth who 
were entering fifth or sixth grade and met Higher 
Achievement’s admission criteria 1 completed a 
standardized test and were surveyed (as were their 
parents) . Then a lottery was used to determine 
which students would be offered the opportunity to 
participate in Higher Achievement (i.e., the “treat- 
ment group”) and which would not (i.e., the “con- 
trol group”) . This design ensured that at the start of 
the study the two groups of youth were, on average, 
identical, and that any differences that emerged 
between them over time could be attributed to the 
Higher Achievement program. 2 

After the initial assessment, we re-surveyed both 
groups of youth and their parents several times: 

• In the spring, one, two and four years after ran- 
dom assignment; and 

• In the fall of 2010, when we conducted a survey 
focused primarily on their learning and experi- 
ences over the previous summer. 

At each time point, our surveys measured attitudes, 
behavior and participation in a range of activities. 
Standardized tests were administered, in concert 
with the surveys, to assess youth’s performance in 
reading comprehension and math problem-solving 
(which we refer to throughout this brief as “read- 
ing” and “math”). In the fourth-year follow-up 
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The study has yielded several other publications: 

Testing the Impact of Higher Achievement’s Year- 
Round Out-of-Schooi-Time Program on Academic 
Outcomes (Herrera et al. 2011a) summarizes findings 
from the first two follow-ups. 

Summer Snapshot: Exploring the Impact of Higher 
Achievement’s Year-Round Out-of-Schooi-Time 
Program on Summer Learning (Herrera et ai. 201 1 b) 
describes the program’s effect on learning during the 
summer of 201 1 . 

Findings from these reports and the current brief are 
described in a more detailed technical report about 
findings across the entire evaluation (see Linden et 
ai. 2013). 


survey, we also asked parents and youth about the 
high school application process. Finally, in addition 
to these sources of data, we conducted program 
observations, interviews and surveys of Higher 
Achievement program staff, mentors and teachers 
to elicit information about staff qualifications and 
training, the program’s curriculum and culture, and 
other implementation issues. 

The evaluation has yielded two previous reports — 
one describing Higher Achievement’s impacts after 
two years and the other outlining program effects 
over the course of one summer (see textbox) . This 
brief summarizes the findings from the full four-year 
evaluation, addressing the following core questions: 

• What is Higher Achievement? 

• Who does the program serve? 

• How long do youth participate? 

• Does Higher Achievement improve academic 
performance over one, two and four years? 

• Does Higher Achievement affect high school 
placement? 

• If so, how does the program foster these benefits? 

We conclude with a discussion of the implications 
of these findings for funders, policymakers and 
program leaders, as well as possible directions for 
future research. 


What Is Higher Achievement? 

Higher Achievement is an intensive summer and 
after-school program that began in its current 
form in 1999 in Washington, DC. 3 Today, there are 
Higher Achievement programs in Washington, DC/ 
Alexandria, VA; Richmond, VA; Pittsburgh, PA; and 
Baltimore, MD. 4 The study includes the five Higher 
Achievement centers that were operating in DC and 
Alexandria when the study began. 5 

Each center serves about 85 students, or “scholars,” 
recruited mainly through school referrals. Starting 
the summer before youth enter fifth or sixth grade, 
and extending through eighth grade, Higher 
Achievement provides scholars with up to 650 hours 
of academic instruction per year, as well as enrich- 
ment activities and targeted, academic mentoring. 
See the “Higher Achievement’s Afterschool and 
Summer Academies” textbox on the next page for 
more details about the program. 

Who Does Higher Achievement Serve? 

A total of 952 youth participated in the study. They 
were recruited in three annual cohorts starting in 
2006. 6 A little over half of these students entered 
the study as rising fifth graders. The remainder 
started in the summer before the sixth grade. 

Most of the students were African American (75 
percent); 13 percent were Latino. More than half 
(59 percent) were girls. About 60 percent were 
eligible for free or reduced-price lunch, and only 
28 percent lived with both parents. Twelve percent 
reported speaking a language other than English 
at home. Thus, in many ways, the youth were fairly 
typical of disadvantaged students in the DC area. 

Yet, they also differed in important ways. First, 
these youth scored slightly above the national aver- 
age on the standardized tests — much better than 
the typical disadvantaged student in the DC area. 7 
Second, they seemed to have better-than-average 
grades. Close to two thirds (65 percent) of applying 
youth reported earning mostly As or As and Bs in 
school, and only 6 percent reported earning mostly 
Cs or lower. Additionally, to be eligible for Higher 
Achievement, students and their parents needed 
to complete an application, attend an in-person 
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Higher Achievement’s Afterschool and Summer Academies 

Higher Achievement is a year-round out-of-school-time (OST) program with two main components, the Afterschool 
Academy and the Summer Academy, both of which participating scholars are required to attend. 

During the school year, Higher Achievement’s Afterschool Academy is offered three days a week, from 3:30 p.m. to 8 p.m., 
over a 25-week period. Each daily session includes homework help, dinner, an arts or recreation elective, a 25-minute 
“community gathering” (attended by all staff, volunteers and youth) and 75 minutes of small-group academic instruction 
using a structured curriculum. The small groups are led by volunteer mentors who work with their group one day a week 
and commit to staying with the program through at least one school year. One day each week is focused on math, one 
on literature and one on a seminar that includes such topics as creative writing, conflict resolution and technology. The 
curriculum that guides these mentoring groups follows skill standards set by the DC and Virginia public school systems 
and reflects Common Core state standards that are being developed and adopted across the country. It is framed around 
four social justice themes (freedom, justice, solidarity and voice) and is designed for youth’s specific grade in school. The 
Afterschool Academy also offers scholars the opportunity to take part in monthly field trips, career shadowing days and 
community service projects. 

Higher Achievement’s six-week Summer Academy operates from 8 a.m. to 4 p.m., five days a week. Scholars have 
four core classes, focusing on academic concepts they will be exposed to during the coming school year. Classes are 
taught by paid teachers in math, science, social studies and literature and typically include about 13 youth. As with the 
Afterschool Academy, the summer curriculum is aligned with the DC and Virginia public school standards and assess- 
ments. In addition to core academics, youth take two electives (such as sculpture, chess or martial arts) and participate in 
weekly field trips, academic competitions and a three-day out-of-town university trip, in which they experience college life 
by sleeping in dorms, eating in the cafeteria and attending lectures. 

As students approach high school, Higher Achievement increases its focus on this important transition. For example, at 
the end of the seventh grade, staff hold a “family night” that includes a group discussion with parents about high school 
applications. During the Summer Academy, rising eighth graders spend two days visiting high-quality high schools and 
engaging in guided discussions about these schools. Staff also meet individually with each eighth grade parent to review 
the student’s grades, answer questions and make recommendations for high school. During the following fall, eighth grad- 
ers have one mentoring session each week devoted to high school applications. The manager of school placement from 
the DC Metro Office also makes biweekly visits to each center to answer individual scholars’ questions and help with the 
application process. 


interview and commit to active participation for 
three to four years. These requirements generated a 
group of eligible youth who had backgrounds simi- 
lar to those of their peers, but who were particularly 
motivated to succeed in school. 

One might ask whether these students need the kind 
of intensive support that Higher Achievement pro- 
vides. In considering this question, it is important to 
note that even highly motivated students can become 
disengaged from school during the middle grades, 
and low-income youth of color may be particularly vul- 
nerable to subsequent academic declines (Simmons 
et al. 1991; see Gutman, Midgley 2000). Moreover, 
because the Higher Achievement applicants were 
already doing fairly well in school, it is unlikely that 
their teachers would have given them the extra help 
or attention needed to raise their achievement lev- 
els to be on par with those of the high-performing 


students who typically attend DC’s more competitive 
schools. Thus, these youth could very well benefit from 
the types of support Higher Achievement offers. 

This study allows us to determine whether, in fact, 
they do. It examines the difference that a program 
like Higher Achievement can make, over several 
years, for this group of initially motivated students, 
compared to an equally motivated control group. By 
comparing the outcomes of the eligible youth who 
were offered a spot in the program with the outcomes 
of those who were not, we were able to discern Higher 
Achievement’s effects on youth’s academic trajec- 
tories. Examining differences in the experiences of 
these two groups of youth, in turn, sheds light on how 
these effects may have been achieved and the types of 
support that could promote similar gains for promis- 
ing young people in other disadvantaged communi- 
ties around the country. 
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How Long Do Youth Participate? 

Higher Achievement is a multiyear program with a 
curriculum that advances from grade to grade and 
many activities that are designed for specific age 
groups. Youth are typically admitted only at the end 
of the fourth or fifth grade, when they can attend 
the full three- or four-year program, and they are 
asked to commit to that full program. (Scholars who 
enter as rising fifth graders can receive four years 
of programming before graduating from Higher 
Achievement; scholars who enter as rising sixth grad- 
ers can receive three years of programming. ) 

Keeping youth engaged in OST programs over time 
is challenging. Few programs have been able to con- 
sistently retain youth for more than a year or two, 
particularly during middle school. For example, a 
survey of 198 OST programs in six cities found that, 
on average, programs retained only 22 percent of 
their middle-school-aged participants for a year or 
more (Deschenes et al. 2010). 8 

For these reasons, it is important to know how 
many youth actually stay in Higher Achievement for 
the entire length of the program. At the first-year 
follow-up, 75 percent of the youth in the treatment 
group who completed our survey reported that 
they were still attending Higher Achievement. After 
two years, this percentage was still fairly high, at 70 
percent. Four years after the baseline, when con- 
sidering only those youth who started the program 


before the fifth grade (i.e., those who were still 
eligible to participate four years later) , 47 percent 
were still attending. Even acknowledging that these 
numbers are likely slightly inflated (because youth 
who completed the survey were probably more 
likely to be involved with Higher Achievement 
than those who did not) , these retention rates are 
impressive, especially compared to those seen in 
other OST programs serving middle school youth. 

Does Higher Achievement Improve 
Academic Performance? 

Retaining youth over time provides a good foun- 
dation for making a difference in their lives, but 
not all OST programs — even those that effectively 
engage youth — are capable of producing measur- 
able academic gains. We assessed youth’s academic 
progress via standardized tests of math and reading, 
conducted at each follow-up. 

After one year, youth in the treatment and control 
groups performed similarly. However, after two years, 
youth with access to Higher Achievement performed 
significantly better on both tests (see Table 1 ) . By 
the fourth-year follow-up, when about half of the 
study youth had transitioned to high school and 
thus no longer had access to the program, Higher 
Achievement youth retained their edge in math, but 
not in reading. 9 


Table 1 

Effects of Higher Achievement on Standardized Test Scores 



First- Year Follow-Up 

Second-Year Follow-Up 

Fourth-Year Follow-Up 

Math problem-solving 

0.03 

0.10* 

0.11* 

Reading comprehension 

0.02 

0.08+ 

0.04 


Note: The values in the table are standardized mean differences, namely “effect sizes." The effect sizes were calculated as the estimated average difference 
between each student’s score and the national average score divided by the standard deviation of the national scores. (The standard deviation is a measure of 
the variation of scores around the average score for the test.) An effect size of .10, for example, means that the treatment group’s average score is 1/10 of a 
standard deviation higher than the control group’s average. These regression estimates control for a number of variables— the youth's baseline scores for read- 
ing and math; self-perceptions of academic abilities, industry and persistence, creativity, enjoyment of learning, curiosity, and ability to change the future through 
effort; peer academic support; and general adult support— in addition to grade at baseline, gender, age, receipt of free or reduced-price lunch, race and whether 
the student applied with a sibling. Family-level controls include fixed effects for the center to which the family applied, parent’s education level, household 
income, household composition and whether English is the primary language spoken at home. 

***p < .001 level of significance. 

** p < .01 level of significance. 

* p < .05 level of significance. 

+ p < .10 level of significance. 
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We also explored whether any specific groups 
of youth may have benefited more than others. 
Generally, Higher Achievement’s effects on aca- 
demic performance did not differ as a function of 
any of the youth characteristics we examined. 10 

How Large Are These Effects? 

To put these gains into perspective, we compared 
them to the typical 12-month gains made by youth 
in the same grades without any intervention. Bloom 
et al. (2008) estimated these gains using national 
spring-to-spring average changes in seven major 
standardized tests. During the middle school years, 
the average yearly change in test scores was .32 in 
reading and .42 in math. Thus, the second-year 
impacts seen in our study (i.e., the extra help 
Higher Achievement provides by the end of the 
second year) are about a quarter of a year’s prog- 
ress for an average middle school student in both 
reading and math. Similarly, the fourth-year math 
impact represents a boost of approximately 26 per- 
cent of what youth this age would typically experi- 
ence in one year. 

Higher Achievement’s impacts are larger than 
those seen in other rigorous, large-scale evaluations 
of OST programs. 11 To date, there have been very 
few large-scale randomized controlled trials of OST 
programs. Two of the largest randomized studies 
examined impacts after two years. James-Burdumy 
et al. (2007) found no academic effects from the 
21 s '- Century Community Learning Centers in a 
large-scale experimental evaluation. In a follow-up 
study that examined the marginal impacts of a well- 
delivered, research-based math and reading cur- 
riculum, above and beyond the typical homework 
help/academic period offered in the 21 st Century 
Community Learning Centers, Black et al. (2009) 
found that the enhanced programs in the sample 
did not significantly increase math scores over the 
typical program and decreased reading scores by .21 
standard deviations. 12 ’ 10 

More comprehensive educational interventions — 
like changing the entire school-day setting for 
youth — have yielded larger impacts than those pro- 
duced by Higher Achievement. For example, Tuttle 
et al.’s (2013) evaluation of KIPP schools found 
much larger four-year impacts in both math (.31) 
and reading (.22). And the Harlem Children’s Zone 


Promise Academy charter schools produced even 
larger improvements in math scores (.23 per year), 
but no improvements in English (Dobbie, Fryer 
2011). These efforts expand well beyond those of 
OST programs, however, both in scope and cost. 

Why Did the Program Yield Consistent 
Improvements in Math but Not Reading? 

Although other rigorous studies of OST pro- 
grams — and of educational interventions more 
broadly — have found improvements in math, 
changes in reading scores are much less common, 
and when effects on reading are found, they are 
generally smaller than those in math (e.g., Decker 
et al. 2004; Dobbie, Fryer 2011; Abdulkadiroglu et 
al. 2011; Furgeson et al. 2012; Tuttle et al. 2013). 
Very few OST programs have rigorous evidence 
of improvements in both areas. Thus, it is impres- 
sive that Higher Achievement was able to improve 
reading scores, even if only in the second year of 
the program. 

These second-year results likely reflect the fact that 
the program has a strong focus on reading and 
literature. One day a week during the school year 
and one class a day during the summer is focused 
on literature, and grammar and reading compre- 
hension are woven into all of these lessons. Higher 
Achievement scholars also participate regularly in 
contests and have many informal learning oppor- 
tunities that may boost their language abilities. For 
example, scholars are given a “word of the week” 
and “quote of the week” in Higher Achievement’s 
community meeting — a gathering that emphasizes 
public speaking. 

Given this focus on reading and language, it is 
perhaps surprising that Higher Achievement did 
not have even stronger and longer-lasting effects 
on reading comprehension. The high motivation 
level of the controls may help to explain this pat- 
tern. Students proficient in reading can fairly easily 
improve their reading comprehension on their own 
(while improving problem-solving independently 
is more difficult) . We observed that at least two 
thirds of both the control and treatment groups had 
“done writing” (e.g., poems, letters or essays) not 
assigned at school, and almost three quarters had 
read books not assigned at school at all three points 
of follow-up. Thus, even without access to Higher 
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Achievement, these youth appeared to be engaged 
in activities conducive to advancing the types of skills 
measured by reading comprehension tests. In addi- 
tion, less than half of the eighth-grade treatments 
were still participating in Higher Achievement at the 
fourth-year follow-up — very likely decreasing our esti- 
mated effects, because the analyses include all youth 
in the treatment group regardless of participation. 14 

In sum, Higher Achievement fosters improvements 
in both math and reading. However, these changes 
take more than a year to materialize, and only the 
math impacts were sustained four years after youth 
enrolled in the program. The size of the gains was 
substantial relative to those yielded in other stud- 
ies of OST programs and year-to-year gains typically 
seen in middle school, yet not as large as those 
seen in some more comprehensive educational 
interventions. 

Does Higher Achievement Affect 
High School Placement? 

In addition to examining Higher Achievement’s 
impact on academic performance, we also wanted to 
know whether the program accomplished its goal of 
improving youth’s matriculation at high-quality high 
schools. The landscape of schools in DC provides a 
range of strong options for students. For example, 
the DC area is home to a host of well-regarded pri- 
vate schools. Many families of means opt for these 
private schools over available public schools, because 
they believe these schools will offer a richer educa- 
tional experience and better outcomes for youth. 

At a national level, private schools tend to have 
smaller class sizes and more rigorous academic pro- 
grams than public schools (Choy 1997). And seniors 
enrolled in private schools are more likely to go to 
college, even after accounting for differing aspira- 
tions, abilities, and socio-economic status (Falsey, 
Heyns 1984). DC also offers some very strong pub- 
lic magnet and public charter schools. Like private 
schools, DC public magnet schools have an applica- 
tion process that includes a review of youth’s grades, 
test scores and writing and an interview, which are 
used to select students for admission. Some charter 
schools also require that students complete a written 
essay, interview and/ or application, but the quality of 
their submission does not determine whether or not 
students are admitted (if more youth apply to a given 
school than it has slots for, a lottery is used to decide 


who gets in) . Still, these hurdles decrease the odds 
that less motivated families will complete the process 
and that youth will end up attending one of these 
charter schools. 

Higher Achievement encourages scholars to apply 
to strong schools of all three types — private, magnet 
and charter. Because these schools vary widely in 
rigor and focus — particularly the public options — the 
program distinguishes among stronger, more com- 
petitive schools and tailors their recommendations to 
each scholar’s interests and strengths. 

To discern Higher Achievement’s effects, we exam- 
ined high school choices and placement at the last 
(fourth-year) follow-up, when youth were finishing 
either their last year of middle school or their first 
year of high school. 15 We focused on four categories 
of schools: DC-area private schools, academically 
“competitive” public magnet and charter schools, 
“noncompetitive” public magnet and charter schools, 
and public high schools in the youth’s neighborhood 
(i.e., public schools that were not magnet or charter 
schools and that youth could attend without apply- 
ing, based on where they lived). 16 We hypothesized 
that Higher Achievement would make it more likely 
that youth would apply to, be accepted to and attend 
private schools and competitive magnet and charter 
schools and less likely that youth would attend non- 
competitive magnet/ charter schools or the public 
school in their neighborhood. 

As seen in Table 2 on the next page, we found that 
parents of youth in the treatment group were indeed 
more likely than those of controls to report that their 
child applied to, was admitted to and matriculated at 
private high schools. About 27 percent of treatments 


How Did We Determine that Schools Were 
Academically “Competitive”? 

For the purposes of this study, we defined competi- 
tive magnet schools as those with an academic focus 
(which were thus academically competitive to get into). 
Competitive charter schools were defined as those 
that the DC public school system had classified as 
“Tier 1 ” in their three-tier classification system (based 
on such measures as standardized test scores and 
graduation rates). Academically noncompetitive char- 
ters and magnets were those that did not fall into the 
“competitive” category. 
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Table 2 

High School Application and Matriculation 



Private Schools 

Competitive 
Public Charter/ 
Magnet Schools 

Noncompetitive 
Public Charter/ 
Magnet Schools 

Neighborhood 
Public Schools 3 

Applied 13 

Controls (%) 

21 

55 

22 

N/A 

Treatment - Control Difference (%) 

6+ 

-1 

-8** 

N/A 

Admitted 15 

Controls (%) 

14 

49 

16 

N/A 

Treatment - Control Difference (%) 

7* 

-3 

-6* 

N/A 

Matriculated 

Controls (%) 

9 

43 

13 

35 

Treatment - Control Difference (%) 

6* 

-4 

_Y** 

5 


Note: Each “Controls” row presents the proportion of youth in the control group who applied to, were admitted to or matriculated at each type of school. These 
values are estimated, controlling for a number of variables— the youth’s baseline scores for reading and math; self-perceptions of academic abilities, industry 
and persistence, creativity, enjoyment of learning, curiosity, and ability to change the future through effort; peer academic support; and general adult support- 
in addition to grade at baseline, gender, age, receipt of free or reduced-price lunch, race and whether the student applied with a sibling. Family-level controls 
include fixed effects for the center to which the family applied, parent’s education level, household income, household composition and whether English is the 
primary language spoken at home. The “Treatment - Control Difference" rows show the proportion of youth in the treatment group who applied to, were admit- 
ted to or matriculated at each type of school above and beyond the percentage of controls. For example, in the first set of cells, 21 percent of controls applied 
to a private school; 6 additional percentage points (27 percent) of treatments applied to a private school. This was a statistically significant difference. 

***p < .001 level of significance. 

** p < .01 level of significance. 

* p < .05 level of significance. 
f p < .10 level of significance. 

a Application to and acceptance at neighborhood public high schools were not calculated because these schools do not require an application. 

Youth could apply and be admitted to more than one type of school. 


applied to these schools, compared to 21 percent of 
controls. Treatment youth were also 50 percent more 
likely than control youth to be admitted (21 percent 
versus 14 percent), and 67 percent more likely to 
matriculate (15 versus 9 percent) at these schools. 

Many students also applied to public charter and mag- 
net schools. In fact, close to two thirds of the youth 
in both the treatment and control groups applied to, 
and close to half eventually chose to attend, a public 
charter or public magnet school. Although treatment 
and control youth matriculated to competitive public 
magnet and charter schools at similar rates, treatment 
youth were much less likely to apply to and matricu- 
late at noncompetitive public magnet/ charter schools 
(6 percent versus 13 percent for controls). Essentially, 
Higher Achievement steered scholars away from 
weaker magnet and charter schools. 


Finally, less than half of youth in both the treatment 
and control groups ultimately decided to attend the 
public high school in their neighborhood. We know 
that some youth moved to very strong neighbor- 
ing school districts over the course of the study (for 
example, in Virginia and Maryland). However, we did 
not have a consistent measure of quality or “competi- 
tiveness” that we could use to categorize the neigh- 
borhood schools, thus we do not know if Higher 
Achievement youth were systemically more likely to 
attend a high-performing neighborhood school than 
were youth in the control group. Additional research 
could test this hypothesis. 

In sum, our findings suggest that Higher 
Achievement did not affect the odds that youth 
would matriculate at a public high school in their 
neighborhood. Yet the program does appear to 
expand the options available to its students by 
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making them more likely to apply to and attend 
private schools and less likely to apply to and attend 
weaker public magnet and charter schools. This, 
in turn, may position youth for better outcomes in 
high school and beyond. 

How Are These Changes Fostered? 

To understand more about how Higher Achievement 
may have produced these benefits for youth, we 
examined a number of potential contributing fac- 
tors. We found that — over the course of the study — 
Higher Achievement fostered differences in the 
treatment group in five of the areas we examined. In 
particular, Higher Achievement changed: 

• The types of academic activities youth engaged in; 

• The extent to which youth engaged in academic 
activities with adults; 

• Youth’s academic attitudes and behaviors; 

• The high school-related activities youth engaged 
in; and 

• The extent to which youth’s parents participated 
in the high school application process. 

Higher Achievement did not change: 

• Youth’s learning over the course of the 
summer; 17 

• The types of peers youth spent time with; 

• The amount of general adult support youth 
experienced; or 

• Youth’s stated preferences for the type of high 
school they wanted to attend. 

We discuss each of these outcomes in more depth 
below. 

Educational Programming 

Academic OST programs aim to improve student 
performance by providing academic experiences 
that add to and complement what youth do dur- 
ing the school day. Higher Achievement may have 
boosted youth’s test scores, at least in part, by sim- 
ply providing more academic instruction and expe- 
riences than youth would have had otherwise. 


To test this hypothesis, we compared the educational 
experiences of scholars and their peers, in terms of 
the total amount of time they spent in academic OST 
programs and their likelihood of engaging in specific 
academic activities outside of school. 18 

First, as would be expected, we found that youth 
in the treatment group spent much more time 
in academic OST programs than their peers. 
Importantly, though, the control group also spent 
a significant amount of time in academic OST 
programs. 19 At the first follow-up, for example, 
they reported spending an average of 1.7 days 
per school week and 0.8 days per week during the 
summer in these programs. However, youth in the 
treatment group received an additional 1.5 days 
of instruction per week during the academic year 
and 2.2 additional days during the summer. This 
translates to about 10 additional hours of instruc- 
tion each week during the academic year and 20 
hours per week during the summer. Significant but 
smaller differences in time spent in academic OST 
programs remained two and four years after base- 
line. Converting these values into annual estimates 
(assuming the 25 weeks during the academic year 
and 6 weeks during the summer when the pro- 
gram operates) , Higher Achievement provided 
youth with about 370 additional educational hours 
in the first year and 327 in the second. Even in the 
fourth year, when participation was the lowest, the 
amount of additional time was still 226 hours. 

Second, we found that youth in the treatment group 
were more likely than their peers in the control 
group to participate in several specific academic 
activities — lor example, participating in academic 
contests or doing writing outside of school assign- 
ments — as well as several other enrichment activities 
(see Table 3 on the next page). At each follow-up, 
treatment youth participated more frequently in 
most of these activities, despite the fact that controls 
were also fairly likely to participate in them. 

Preventing the Summer Learning Loss 

Preventing the decline in academic performance 
that typically occurs over the summer is a common 
goal of OST programs that include summer activities. 
Disadvantaged youth are especially likely to experi- 
ence academic setbacks over the summer (Cooper 
et al. 1996), in part because learning is dependent 
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Table 3: 

Academic and Enrichment Activities 


Activity 3 

First- Year Follow-Up 

Second-Year Follow-Up 

Fourth-Year Follow-Up 


Controls 

<%) 

Treatment 
- Control 
Difference 
(%) 

Controls 

<%) 

Treatment 
- Control 
Difference 
(%) 

Controls 

<%) 

Treatment 
- Control 
Difference 
(%) 

Community service or volunteer work 

53 

0 

56 

4 

70 

4 

Spoken to a group outside of school about your 
ideas/work 

57 

6+ 

55 

9* 

58 

11* 

Visited a college campus to see what it would be 
like to be a college student 

44 

28** 

45 

28** 

53 

25** 

Read books that are not for school 

71 

3 

75 

4 

77 

8+ 

Written things (e.g., poems, letters or essays) not 
assigned at school 

66 

7* 

71 

3 

68 

14** 

Visited a business to see what it would be like to 
work there 

52 

4 

53 

8* 

56 

14** 

Gone to events outside of your neighborhood with 
your OST program 

67 

10** 

74 

6+ 

75 

9* 

Participated in academic contests at your OST 
program 

55 

13** 

57 

11** 

55 

16** 


Note: The values listed in each “Controls” column are the proportions of youth in the control group who reported ever participating in the activities listed. These 
values are estimated, controlling for a number of variables— the youth’s baseline scores for reading and math; self-perceptions of academic abilities, industry 
and persistence, creativity, enjoyment of learning, curiosity, and ability to change the future through effort; peer academic support; and general adult support— in 
addition to grade at baseline, gender, age, receipt of free or reduced-price lunch, race and whether the student applied with a sibling. Family-level controls 
include fixed effects for the center to which the family applied, parent’s education level, household income, household composition and whether English is the 
primary language spoken at home. The values in the “Treatment - Control Difference" columns are the proportions of youth in the treatment group who reported 
participating in the activities, above and beyond the proportions reported by youth in the control group. 

”*p < .001 level of significance. 

** p < .01 level of significance. 

* p < .05 level of significance, 
t p < .10 level of significance. 

a With the exception of the last two activities, youth were asked whether they had participated in these activities broadly (i.e., not only in the context of an 
OST program). 


on home and community resources, which are often 
lacking for poor children. Higher Achievement aims 
to give scholars academic continuity by providing 
intensive instruction during the summer months — 
an approach that could help stem the summer learn- 
ing loss that so many youth experience. 

To evaluate Higher Achievement’s effects on the 
summer learning loss, we conducted assessments in 
the spring before the 2010 Summer Academy and 
again in the fall, after the Academy ended. 20 As of 
the spring, treatment students (who had had the 
opportunity to participate in Higher Achievement 
for two to three years) already had significantly 


better test scores in both reading and math, com- 
pared with youth in the control group. However, 
surprisingly, the following fall, the control students 
had not experienced the expected summer learning 
loss. Rather, their reading scores increased, and their 
math scores remained roughly stable. Scholars’ 
reading scores also increased, but not as much as 
those for the control group; the treatment group’s 
edge in math was also smaller and no longer statis- 
tically significant. As a result, treatment youth did 
not make bigger gains than controls over the sum- 
mer, suggesting that while Higher Achievement may 
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prevent the summer learning loss, the very moti- 
vated youth it serves would have found other ways 
to avoid this decline even without the program. 21 

Changing Academic Attitudes and Behavior 

Higher Achievement hopes to improve scholars’ tra- 
jectories, in part by improving their attitudes toward 
school and learning, and increasing their confidence 
in their own abilities — attitudes that research sug- 
gests are linked with stronger academic performance 
(e.g., Wong et al. 2002). Thus, in addition to assess- 
ing youth’s performance on standardized tests, we 
measured six aspects of students’ self-perceptions: 
industry and persistence (i.e., their tendency to per- 
sist despite failure) , creativity, perceptions of their 
own academic abilities, enjoyment of learning, curi- 
osity, and the extent to which they believe they could 
change the future through their own effort. 

Surprisingly, rather than improving students’ aca- 
demic perceptions, after their first year of access to 
the program, youth in the treatment group were 
less confident relative to controls. After the first 
follow-up, however, these differences basically dis- 
appeared. 22 These negative effects may have been 
the consequence of entering a rigorous program 
with similarly motivated peers — a process that could 
have caused students to realize they had room for 
improvement. While decreased confidence could 
be viewed as an adverse outcome, it is possible that 
this process actually primed youth for future gains, 
encouraging scholars to work harder. 23 

We also assessed the effects of the program on 
students’ reports of their own negative behaviors, 
such as stealing, breaking something on purpose 
or going to the principal’s office. Similar to the dip 
scholars experienced in their academic attitudes, 
students in the treatment group reported worse 
behavior one year after baseline. Although this dif- 
ference became less pronounced in the last two 
follow-ups, it is still noteworthy, echoing reports 
by James-Burdumy et al. (2007) in their evaluation 
of the 21 st Century Community Learning Centers. 
Higher Achievement scholars, in particular, may 
have “acted out” due to the stress of a more com- 
petitive environment. The program also encourages 
personal responsibility and honesty, so these youth 
may have been more apt than their peers to report 
negative behaviors. 24 


Changing Youth’s Peer Group 

Past research has shown that youth’s peer relation- 
ships can also affect academic outcomes (e.g., 
Wentzel, Caldwell 1997). It is possible that by putting 
students in closer contact with similarly motivated 
students, Higher Achievement provides a more 
academically supportive peer group, which might 
in turn encourage more diligent academic work 
and shifts in scholars’ decisions about high school. 
Overall, Higher Achievement did change ivliere schol- 
ars made their close circle of friends — youth in the 
treatment group were more likely to report that they 
met at least one of their closest friends through their 
academic OST program or through their family (and 
less likely to report that they had made these friends 
at school) . 25 However we found no difference across 
groups in how academically supportive they per- 
ceived their peers to be. 

We also asked about four specific academically ori- 
ented activities that youth could have participated 
in with their peers — whether they had helped their 
peers with their school work, presented their ideas 
in front of a group of peers, gotten praise from their 
peers for their achievements or talked with their 
peers about a math or science project outside of 
school. Similarly high proportions (over 80 percent) 
of youth in both groups had engaged in most of these 
activities with peers, at all three time points. 26 

Increasing Youth’s Access to Adult Support 

Higher Achievement provides scholars with 
many opportunities for positive adult interaction. 
Program staff lead centers across multiple years, 
teachers interact with small groups of students 
throughout the Summer Academy, and mentors are 
encouraged to work with their mentoring group 
(two to five scholars) 27 throughout the school 
year and beyond. These relationships could foster 
improvements in academic performance both by 
providing more “emotional” aspects of support 
(e.g., helping youth through personal problems, lis- 
tening, caring about youth) and by providing more 
direct academic support and guidance (e.g., talking 
with youth about college or high school) . We asked 
youth questions about both types of adult support. 

More than half of control students (typically, 
over two thirds) reported engaging with nonpa- 
rental adults in each of the academically related 
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interactions that we asked about (e.g., talking with 
adults about how to get into a good high school, 
going to college or future jobs). Youth with access 
to Higher Achievement were still about 5 to 10 
percentage points more likely to engage in each 
of these activities than controls. However, this 
increased engagement with adults did not trans- 
late into increases in more general perceptions 
of adult support — that is, in the number of adults 
who they said paid attention to them or were avail- 
able to help them with problems. In fact, youth 
in both groups reported similarly high numbers 
of supportive adults outside their family (on aver- 
age, a little over three adults), suggesting that even 
without Higher Achievement, these youth felt rela- 
tively well supported. 


Student Preferences 

To understand more about how Higher Achievement 
may have affected youth’s high school choices, we 
asked youth what characteristics were important to 
them as they decided which schools to apply to and 
attend, including, for instance, the school’s academic 
strength, its proximity to home, its cost, whether 
they believed they had a good chance of getting in, 
whether other friends were attending and whether 
others (e.g., parents, teachers, OST staff) thought 
they should apply. Higher Achievement did not 
appear to affect these priorities. Both control and 
treatment students reported that the school’s aca- 
demic strength was the most important characteristic 
for them, followed closely by the school’s strength 
in the arts, sports or another area of interest. Thus, 


Table 4 

High School Application Activities 


Activity 

Controls 

<%) 

Treatment - Control Difference 
(%) 

Attended a mock interview 

15 

12** 

Attended a test preparation class for the SSAT or HSPT a 

13 

11** 

Practiced for the SSAT or HSPT, but not as part of a class 

12 

17** 

Took the SSAT or HSPT 

18 

7* 

Applied for a scholarship 

13 

6t 

Received a scholarship 

10 

5 t 

Visited a high school of interest 

60 

7t 

Spoke with teachers or other staff at a school of interest 

52 

10* 

Spoke with students who attended these schools about how 
they liked it there 

54 

12** 

Got information about specific high schools 

60 

4 

Attended a “shadow” day at a high school 

26 

5 


Note: The values listed in the “Controls” column are the proportions of youth in the control group who reported ever participating in the activities listed. These 
values are estimated, controlling for a number of variables— the youth’s baseline scores for reading and math; self-perceptions of academic abilities, industry 
and persistence, creativity, enjoyment of learning, curiosity, and ability to change the future through effort; peer academic support; and general adult support— in 
addition to grade at baseline, gender, age, receipt of free or reduced-price lunch, race and whether the student applied with a sibling. Family-level controls 
include fixed effects for the center to which the family applied, parent’s education level, household income, household composition and whether English is the 
primary language spoken at home. The values in the “Treatment - Control Difference" column are the proportions of youth in the treatment group who reported 
participating in the activities, above and beyond the proportions reported by youth in the control group. 

***p < .001 level of significance. 

** p < .01 level of significance. 

* p < .05 level of significance, 
t p < .10 level of significance. 

a The Secondary School Admission Test (SSAT) and High School Placement Test (HSPT) are tests required for admission to many private and 
independent schools. 
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even without exposure to the program, this group of 
motivated students had a strong preference for aca- 
demically rigorous schools. 

High School-Related Activities 

Although Higher Achievement did not appear to 
influence youth’s high school priorities and prefer- 
ences, the program did boost youth’s engagement 
in a wide range of activities related to the high 
school application process (see Table 4 on the 
previous page). For example, youth in the treat- 
ment group were more likely to visit schools and to 
speak with students or teachers from a high school 
of interest. Higher Achievement also increased the 
likelihood that students took steps in the applica- 
tion process for more competitive high schools. For 
instance, a higher percentage of treatment students 
attended a mock interview, prepared for and took 
the admissions test, and applied for and received 
a scholarship, compared to youth in the control 
group. 

Parent Involvement 

We also found some evidence that Higher Achieve- 
ment increases parent involvement in the appli- 
cation process, although this evidence is not as 
strong as that presented for the other experiences 
described above. The parents of youth in the treat- 
ment group were significantly more likely than 
parents of control youth (67 versus 59 percent 
respectively) to report that they (as opposed to 
their child) spent the most time on their child’s 
high school applications. Treatment youth were also 
more likely than controls to indicate that their par- 
ents provided help in test preparation and applying 
for financial aid. 28 

Higher Achievement strives to involve parents 
throughout the middle school years, but it par- 
ticularly focuses on engaging them in high school 
placement activities as scholars approach this 
important transition. At the fourth-year follow-up, 
almost two fifths (37 percent) of parents whose 
children were still attending Higher Achievement 
reported that they spoke to program staff about 
their child at least once a week. Staff work with 
parents as a group and individually to answer ques- 
tions, review each student’s grades and make spe- 
cific high school recommendations. This process 
may give parents tools they need to take a more 
active role in their child’s applications. 


Conclusions 

Middle school is a key juncture in a student’s aca- 
demic path. Yet, there are relatively few OST pro- 
grams serving middle-school-aged youth, and most 
have not figured out how to effectively engage these 
youth for more than a day or two a week (James- 
Burdumy et al. 2007; Grossman et al. 2002). This is 
mainly because this age group is very difficult to serve. 
Early adolescence is a time of powerful physical, 
cognitive and emotional changes. Peers and adults 
outside the home become more central to youth, and 
they seek more autonomy and control over their lives, 
including in their learning environments. 

Regrettably, most middle schools are organized 
in ways that do not meet these emerging develop- 
mental needs. For example, public middle school 
classrooms are generally organized as teacher-led, 
whole-class instruction, giving students little control 
over their learning (Newman et al. 2000; Holcomb- 
McCoy 2007) . Students also rotate between classes, 
making it difficult to form solid relationships with 
their teachers. Consequently, many students — even 
the very able — become disengaged from learning 
during the middle school years. By contrast, Higher 
Achievement’s Afterschool and Summer Academies 
center around small-group instruction led by 
volunteer mentors (during the school year) and 
trained teachers (during the summer). This setting 
makes learning an active process, in which students 
can exert more control and provide more input. 

The small-group setting also fosters relationships 
between students and their mentors and teachers. 

This evaluation provides rigorous evidence that 
intensive OST programs like Higher Achievement 
can produce results. The program successfully 
engages middle school youth and retains siz- 
able numbers of them for multiple years. Higher 
Achievement not only improves their academic 
performance, but also increases the odds that 
students will attend a private high school and 
decreases their likelihood of attending a weaker 
public magnet/ charter school. The study’s find- 
ings highlight a number of important lessons for 
policymakers, funders and OST program leaders: 

Intensive OST programs can boost critical math 
skills. Higher Achievement improved both math 
and reading by the end of the second summer and 
school year, with gains in math persisting after four 
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years. And these gains are larger than those yielded 
in other large-scale, rigorous studies of OST pro- 
grams. The impact in math is particularly important 
because recent work has shown that math skills in 
middle school and ninth grade are very predictive of 
success in high school, particularly graduation rates 
(Kieffer, Marinell 2012). Thus, OST programs that 
improve math skills may help put youth on a positive 
path that lasts well beyond the program itself. 

While Higher Achievement’s impacts are impressive 
relative to others in the OST field, they are not as 
large as those yielded by some whole-school mod- 
els, such as KIPP schools or the Harlem Children’s 
Zone Promise Academies. To produce these kinds 
of large improvements in test scores, funders and 
policymakers will likely need to support more com- 
prehensive educational interventions. Our evalua- 
tion shows that programs like Higher Achievement 
are a good option, short of these more expensive, 
whole-school reforms. 

Programs like Higher Achievement can help stu- 
dents effectively navigate the high school choice 
process. Perhaps even more important than Higher 
Achievement’s academic impacts were the informa- 
tion and experiences it provided to students about 
college, careers and — most pressing — high school 
choice. The program offers opportunities to talk with 
people from different professions, to job shadow and 
to learn about college and college life, as well as pro- 
viding concentrated support throughout the process 
of selecting and applying to an academically rigorous 
high school. Our study suggests that these activities 
may contribute to youth’s successful matriculation at 
high-quality high schools that, for many, would have 
been out of reach without Higher Achievement. 

Changing youth’s high school trajectories is difficult. 
While the treatment students had better test scores 
than controls over the course of middle school, 
other aspects of the program likely drove the impacts 
around high school choice. Disadvantaged fami- 
lies often have inadequate information about the 
high school application process (Sattin-Bajaj 2009). 
Higher Achievement helped youth take the many 
needed steps to go to a competitive high school, 
including preparing for the high school entrance 
exams, attending a mock interview and applying for 
and receiving scholarships. 


Higher Achievement also recognizes that parents 
are an integral part of improving youth’s academic 
trajectories. If parents are not willing to make the 
commitments necessary to enable their children to 
attend competitive high schools, it will not happen. 
But engaging parents is no easy task — it is some- 
thing that most programs struggle with. Higher 
Achievement successfully involves parents through- 
out middle school and increases their participation 
in the high school application process — efforts that 
likely contributed to the effects we measured. 

OST programs may be most effective when tak- 
ing a long-term approach. Higher Achievement 
represents a comprehensive, long-term investment 
in children’s lives. Its intensive and rigorous aca- 
demic environment improved scholars’ academic 
performance — but not until after the first year of 
program involvement. Similarly, the decision to 
attend a competitive high school is not simply made 
in the eighth grade but rather is a culmination of 
experiences and choices (such as course selection) 
throughout the middle school years (Eccles et al. 
2004). Thus, the fact that Higher Achievement is 
able to work with and support youth throughout 
middle school is probably critical to its success. 

These effects may also compound over time, even 
after program participation ends. Many of the 
youth in the treatment group are now attending 
academically rigorous high schools better prepared 
than they would have been otherwise. These stu- 
dents (as well as those who matriculated to less 
competitive schools but were likewise better pre- 
pared academically) may well end up attending 
better colleges and ultimately having higher-paying 
jobs and careers. They and their parents are now 
familiar with a long set of choices and steps that are 
similar to those they will face in the college applica- 
tion process. Funders and policymakers who hope 
that OST programs can foster substantial changes 
in youth’s trajectories may need to consider longer- 
term investments — like Higher Achievement — to 
produce benefits similar to those seen in this study. 

Higher Achievement’s pattern of effects highlights 
the importance not only of investing in long-term 
programs but also in designing and supporting 
evaluations that last multiple years. Many OST 
programs have undergone evaluations that span a 



Staying On Track: Testing Higher Achievement’s Long-Term Impact on Academic Outcomes and High School Choice 


18 


single year, hoping for academic impacts, and few 
have found them. Raising students’ test scores may 
simply take longer than a year in most OST settings. 

Final Thoughts 

This study points to several important implications 
for policy and research. First, Higher Achievement 
produced these benefits for a group of highly 
motivated but disadvantaged youth. Perhaps the 
program would have yielded even bigger impacts 
if less motivated students — with more pressing 
academic needs and fewer supporting resources — 
were served at the same intensity. 29 However, less 
motivated students and their families might not 
persist in the program long enough for it to have 
its effects. In the end, only another impact study 
could determine with certainty if the program is 
equally (or more) effective for less motivated youth. 
Given the potential long-term benefits of Higher 
Achievement’s model, this is a crucial next-step 
question for research. 

Second, while we have been able to identify some 
of the elements of Higher Achievement that likely 
contributed to youth’s gains, we do not know which 
are necessary and which (if any) could be stream- 
lined without diminishing the effectiveness of the 
program. Future research that rigorously examines 
the mechanisms underlying OST program ben- 
efits — starting with some of those outlined here — 
would provide important guidance about how to 
get the greatest return on OST investments. 

Higher Achievement costs about $4,500 per 
scholar per year. These funds support up to 650 
hours of structured activities over the school year 
and summer (at about $7 per hour) , led by well- 
trained staff who use a curriculum that mirrors and 
builds on what youth learn during the school day. 
Programming related to the high school choice 
process begins in early middle school and ramps up 
considerably as students approach and enter eighth 
grade. This constellation of services appears to keep 
students on a positive path during a critical time 
in their development. Other intensive academic 
OST programs with a similar structure and focus 
may yield similar results and could therefore be a 
viable tool for helping disadvantaged youth stay on 
track during the middle school years and beyond. 
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Endnotes 


1. To enroll in the program, youth must complete an applica- 
tion, attend an interview alone and with their parents, and 
be deemed “academically motivated” by Higher Achievement 
staff (i.e., staff determine that the student understands the 
program’s requirements and is excited about getting involved). 
Parents are also required to attend a program orientation if 
their child is accepted. 

2. The study uses an intent-to-treat design — the most rigorous 
and conservative approach available to researchers — whereby, 
at each follow-up, we surveyed all youth in the sample whom 
we could contact, including all treatment youth regardless 

of whether they had actually attended the program. We then 
grouped youth according to their original random assignment 
status. In the analysis, the outcomes of the originally assigned 
treatment group are compared to those of the originally 
assigned control group. Thus, technically, the study addresses 
the question of whether having access to the program affects 
youth outcomes, not whether program participation, per se, 
affects outcomes. In this way, the impacts we report likely under- 
state Higher Achievement’s effects on those youth who fully 
participate in the program. 

3. An earlier version of the program operated from 1975 to 1998 
and provided some of the services included in the current pro- 
gram exclusively to gifted and advanced students. 

4. City offices are supported by a national office in DC. In each 
city, one or more Achievement Centers is located in an elemen- 
tary or middle school. 

5. The center in Alexandria started serving youth in 2006; we 
began recruiting applicants to that center one year after the 
start of the study. DC also has one additional center that started 
operating in 2010, after recruitment for the study was com- 
pleted. That center is not included in the evaluation. 

6. These cohorts included 277, 276 and 399 youth respectively. 

7. While impressive for this group of disadvantaged youth, it 
should be noted that students who typically matriculate to com- 
petitive public and private schools score even higher. 

8. Some of the programs included in this study were short-term by 
design. While most OST programs encourage youth to partici- 
pate for multiple years, many do not ask youth for a long-term 
commitment and instead structure activities as voluntary 8- to 
12-week or school-year-long sessions, which youth may have the 
option of repeating. 

9. We also conducted these analyses attempting to isolate the 
effects of attending Higher Achievement, rather than simply 
being offered the opportunity to attend the program. The pat- 
tern of effects is very similar to the intent-to-treat estimates, but 
the effects are slightly larger (see Linden et al. 2013). 


10. These tests compared groups of youth that varied on test scores, 
race/ethnicity, income, gender, whether they started the pro- 
gram as rising fifth or sixth graders, the “ward” or area of DC in 
which they lived, whether the youth received free or reduced- 
price lunch, or the Higher Achievement center of application. 

11. It is important to note in this comparison that few published 
studies have tracked students longer than a year. Thus, the 
effect sizes described in the literature typically reflect only one 
year of dosage, whereas the benefits yielded in this study reflect 
two to four years of access to the program. 

12. There are a number of other studies in the literature that use 
random assignment. However, the vast majority have very small 
sample sizes — often fewer than 100 youth (Beckett et al. 2009). 
Chaplin and Capizzano (2006) conducted a randomized con- 
trolled study on a larger sample to estimate the effects of the 
Build Educated Leaders for Life (BELL) summer program on 
students’ reading comprehension scores. However, the control 
group experienced more total days of school than the treat- 
ment group. Their preferred 0.08 treatment effect estimate 
adjusts for the days of school received by each student. The 
unadjusted intent-to-treat estimates show no effect. 

13. Several meta-analyses have reported larger effects on students’ 
standardized test scores. While some of these estimates are com- 
parable to the estimates in our study (Lauer et al. 2006, for exam- 
ple), these meta-analyses (a) rely, in part, on nonexperimental 
studies or experimental studies that use very small samples to 
calculate their overall treatment effect and (b) are subject to 
potential problems associated with aggregating results across 
studies, including, for instance, publication bias. That is, studies 
are often published only when they yield significant findings; 
thus, estimates that aggregate across these studies do not adjust 
for the many studies that have no findings, and, for that reason, 
these estimates are likely somewhat inflated. 

14. One might also suspect that the decay from the second-year to 
the fourth-year follow-up was related to the fact that at the lat- 
ter time point the ninth graders in the sample (i.e., those who 
started the program as rising sixth graders) no longer had access 
to the program’s resources and support. If this were the case, 
then we would expect to see bigger treatment effects for eighth 
relative to ninth graders at the fourth-year follow-up, because the 
eighth graders still had access to Higher Achievement. However, 
there were no significant differences between the reading 
impacts experienced by the eighth and ninth graders. 

15. When they completed their survey in the spring, some parents 
of eighth graders did not know which high school their child 
would attend. A small number, for example, had not heard back 
from all of the schools they had applied to. We followed up with 
these families again over the summer to get their final decision. 
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16. Students have the option of enrolling in a “neighborhood 
school” that is located in a neighborhood other than their own, 
if that school has an available slot. 

17. Only the summer of 2010 was examined in these analyses — the 
second or third summer after youth’s enrollment. 

18. At the first- and second-year follow-ups, these analyses include 
all youth. However, for the fourth-year follow-up, we include 
only students entering the study prior to the fifth grade to 
restrict the sample to those still eligible to attend Higher 
Achievement. 

19. None of the controls attended Higher Achievement. 

20. These assessments included all students who were eligible for 
the 2010 Summer Academy — i.e., all students in Cohort 3 and 
those in Cohort 2 who had applied as rising fifth graders (those 
who had applied as sixth graders were no longer in middle 
school and thus no longer had access to Higher Achievement 
programming by the summer of 2010). 

21. We examined these changes over the course of only one sum- 
mer, after scholars had participated in the program for two or 
three years. We did not test whether the program fosters relative 
improvements over the course of earlier (or later) summers of 
program involvement. 

22. Except for self-perceptions of academic abilities, which showed 

a positive treatment effect in the second year, all of the estimates 
are small and not statistically significant after the first year. 

23. Not all treatment youth experienced this dip in attitudes after 
the first year of the program. It was driven primarily by the 
students entering the study as rising fifth (as opposed to rising 
sixth) graders. The academic attitudes of the fifth grade treat- 
ments fell sharply after their first year in the program — unlike 
their fifth grade control peers, who did not experience this 
decline. This decline is similar to what the later-enrolling youth 
in both the treatment and control groups experienced during 
their sixth grade year — attitudes in both groups fell during the 
sixth grade — which is consistent with a dip that most students 
experience upon entering middle school (Eccles, Midgley 
1989). Unlike the fifth grade treatments, fifth grade control 
students experienced improvements in their attitudes over the 
last year of elementary school. They then declined sharply in 
their first year of middle school. By placing fifth grade students 
in a more competitive environment, the Higher Achievement 
program seems to have caused the middle school decline a year 
before these students would otherwise have experienced it. 


24. We hypothesized that in the fourth year, youth who were now 
in more academically rigorous high schools may have been 
exposed to more strict rules than those they experienced when 
attending their previous public middle school. Thus, these 
treatment youth may have reported more school-related misbe- 
havior (e.g., visits to the principal’s office) than controls who 
were less likely to attend these schools. However, we did not 
find evidence to support this hypothesis: When we conducted 
the analyses separately for the younger and older youth, we did 
not see differences depending on whether the youth were in 
their last year of middle school or first year of high school. 

25. Students could select three options for where they met their 
close friends: school, family (e.g., cousin or sibling) or another 
place. When they responded with “another place,” they were 
asked to specify the location. “An OST program” and “Higher 
Achievement” were self-reported answers tabulated from the 
last category. For Higher Achievement, we checked students’ 
responses by matching the names of listed friends to the stu- 
dents in our sample. 

26. We found only one difference between treatments and controls 
in these four measures across the three time points: At the last 
follow-up, treatment youth were more likely to report that they 
had talked with peers about a math or science problem outside 
of school (85 percent of treatments versus 72 percent of con- 
trols reported this activity) . 

27. The number of scholars in each group decreased over time as 
the study progressed — from four or five in the first couple of 
years to two or three later in the study. 

28. Although treatments were more likely to rate their parents as 
engaging in these two activities, there were eight other applica- 
tion activities that showed no difference between treatments 
and controls, and an aggregate measure across all of these 
activities was not significant. Thus, the differences we estimated 
between these two activities may have been significant by 
chance alone. 

29. In fact, Higher Achievement’s new strategic plan emphasizes 
finding ways to serve less motivated students. 
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